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MEMORANDUM  FOR  ASSISTANT  SECRETARY  OF  THE  NAVY  (FINANCIAL 

MANAGEMENT) 

ASSISTANT  SECRETARY  OF  THE  AIR  FORCE 

(FINANCIAL  MANAGEMENT  AND  COMPTROLLER) 
INSPECTOR  GENERAL,  DEPARTMENT  OF  THE  ARMY 


SUBJECT:  Audit  Report  on  Acquisition-Type  Lessons-Learned  Programs  Within  the 
Military  Departments  (Report  No.  93-173) 


We  are  providing  this  report  for  your  information  and  use.  The  report  discusses 
the  use  of  lessons-leamed  information  during  the  development  and  modification  of 
major  weapon  systems.  The  report  also  discusses  the  attempt  to  establish  an  integrated 
lessons-learned  exchange  program  within  the  Military  Departments.  Comments  on  the 
draft  of  this  report  were  required  by  August  24,  1993;  however,  comments  were  not 
received  from  the  Joint  Logistics  Commanders.  DoD  Directive  7650.3  requires  that  all 
audit  recommendations  be  resolved  promptly.  Therefore,  we  request  that  the  Joint 
Logistics  Commanders  provide  comments  on  the  findings  and  recommendations  by 
November  29,  1993.  Also,  Army  and  Navy  comments  are  not  fully  responsive  to  the 
recommendations.  We  request,  therefore,  that  the  Assistant  Secretary  of  the  Army 
(Research,  Development  and  Acquisition)  and  the  Assistant  Secretary  of  the  Navy 
(Research,  Development  and  Acquisition)  provide  comments  on  the  unresolved  portions 
of  the  recommendations  by  November  29,  1993. 

Recommendations  are  subject  to  resolution  in  accordance  with  DoD  Directive 
7650.3  in  the  event  of  nonconcurence  or  failure  to  comment.  We  ask  that  your 
comments  also  indicate  concurrence  or  nonconcurrence  with  the  internal  control 
weaknesses  highlighted  in  Part  I.  The  report  identifies  no  quantifiable  monetary 
benefits;  Appendix  B  summarizes  potential  benefits  of  the  audit. 

We  appreciate  the  courtesies  extended  to  our  audit  staff.  If  you  have  questions 
on  this  audit,  please  contact  Mr.  James  Koloshey,  Program  Director,  at  (703)  614-6225 
(DSN  224-6225)  or  Mr.  Eddie  Ward,  Project  Manager,  at  (703)  614-6222  (DSN)  224- 
6222).  Appendix  D  lists  the  distribution  of  this  report. 


Robert  f.  Lieberman 
Assistant  Inspector  General 
for  Auditing 


Office  of  the  Inspector  General,  DoD 


Audit  Report  93-173  September  27,  1993 

(Project  No.  2AG-5007) 

ACQUISITION-TYPE  LESSONS-LEARNED  PROGRAMS  WITHIN 
THE  MILITARY  DEPARTMENTS 


EXECUTIVE  SUMMARY 


Introduction.  Lessons-leamed  data  are  recorded  experiences  of  value  to  the  conduct 
of  current  and  future  acquisition  programs.  They  are  conclusions  drawn  from  analysis 
of  feedback  on  deployed  systems  or  systems  currently  under  development  and 
modification.  The  use  of  lessons-learned  data  should  assist  program  offices  in  the 
acquisition  process  by  avoiding  the  repetition  of  mistakes  or  profiting  from  examples  of 
positive  results. 

Objectives.  The  audit  objectives  were  to  determine  whether  weapon  system 
supportability  problems  were  accurately  reported  by  operational  commands  and  whether 
lessons-learned  data  were  adequately  considered  by  acquisition  officials  during  weapon 
system  development  and  modification.  To  accomplish  both  objectives,  \ve  evaluated 
policies  and  procedures  used  by  program  offices  for  identifying  and  using  lessons- 
learned  data. 

Audit  Results.  This  report  contains  two  findings  related  to  the  Military  Departments' 
use  of  lessons-learned  data  during  the  acquisition  process. 

o  The  Military  Departments  did  not  fully  use  or  exchange  formalized  lessons- 
learned  data  during  the  development  or  upgrade  of  major  weapon  systems  (Finding  A). 

o  An  attempt  to  establish  a  lessons-learned  exchange  program  among  the 
Military  Departments  has  not  progressed  much  since  program  inception  in  1989 
(Finding  B). 

Consequently,  acquisition  officials  have  not  realized  the  benefits  of  these  historical  data 
to  avoid  mistakes  or  profit  from  positive  results. 

Internal  Controls.  The  audit  identified  internal  control  weaknesses  as  defined  by 
Public  Law  97-255,  Office  of  Management  and  Budget  Circular  A- 123,  and  DoD 
Directive  5010.38.  They  are  highlighted  in  Part  I.  Policies  and  procedures  were  not 
adequate  to  ensure  that  acquisition  officials  used  the  Military  Departments'  formalized 
lessons-learned  data.  (Finding  A). 

Potential  Benefits  of  Audit.  The  primary  benefits  that  will  be  realized  from 
implementing  the  audit  recommendations  are  unquantifiable  cost  avoidance  through 
better  system  design  and  reduced  system  life-cycle  costs.  These  potential  benefits  are 
summarized  in  Appendix  B.  We  are  not  able  to  quantify  the  savings  related  to  cost 
avoidance  since  future  use  of  weapon  systems  could  not  be  projected. 

Sununary  of  Recommendations,  We  recommended  that  the  Assistant  Secretary  of  the 
Army  (Research,  Development  and  Acquisition)  and  the  Assistant  Secretary  of  the 
Navy  (Research,  Development  and  Acquisition)  strengthen  existing  lessons-learned 
policies  to  ensure  program  offices  use  these  data  during  their  decisionmaking.  We 


recommended  that  the  Army  Deputy  Chief  of  Staff  for  Logistics  establish  a 
computerized  data  base  that  includes  acquisition  and  logistics  lessons-learned  data. 
Also,  we  recommended  that  the  Commander,  Air  Force  Materiel  Command,  enforce 
existing  policies  to  require  the  use  and  dissemination  of  lessons-learned  data  by 
acquisition  offices.  Finally,  we  recommended  that  the  Joint  Logistics  Commanders 
designate  an  executive  agency  that  will  be  responsible  for  program  oversight  and  for 
establishment  of  more  specific  program  guidelines. 

Management  Comments.  The  Army  agreed  with  the  recommendations  addressed  to 
them,  but  did  not  explicitly  address  actions  to  require  program  offices  to  use  formalized 
lessons-learned  data  during  development.  The  Navy  partially  concurred,  stating  that 
policy  requiring  the  use  of  lessons-learned  data  should  be  set  by  DoD.  Navy  will 
advise  program  managers  of  the  availability  of  data  from  existing  reporting  systems, 
but  these  are  not  lessons-learned  data  bases.  The  Air  Force  concurred  with  the 
recommendation  addressed  to  them.  Joint  Logistics  Commanders'  comments  were  not 
received  within  the  specified  time,  but  the  Army,  which  is  the  lead  Military 
Department  in  this  instance,  will  coordinate  a  response  to  the  final  report.  We  request 
the  responses  by  November  29,  1993. 
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Part  IV  -  Management  Comments 

Department  of  the  Army  Comments 
Department  of  the  Navy  Comments 
Department  of  the  Air  Force  Comments 


The  Acquisition  Management  Directorate,  Office  of  the  Assistant  Inspector  General  for 
Auditing,  DoD,  prepared  this  report.  Copies  of  the  report  can  be  obtained  from  the  Secondary 
Reports  Distribution  Unit,  Audit  Planning  and  Technical  Support  Directorate,  (703)  614-6303 
(DSN  224-6303). 


Part  I  -  Introduction 


Background 


Program  officials  responsible  for  major  weapon  system  development  or  upgrade 
should  deploy  systems  that  are  supportable  and  sustainable.  The  identification 
and  utilization  of  lessons-learned  data  during  the  acquisition  process  can 
contribute  to  reduced  weapon  system  life-cycle  costs  while  also  achieving 
readiness  requirements. 

DoD  Instruction  5000.2,  "Defense  Acquisition  Management  Policies  and 
Procedures,"  February  23,  1991,  does  not  specifically  require  the  use  of 
lessons-learned  data  by  acquisition  officials.  However,  Part  6,  Section  C  of  the 
Instruction  does  emphasize  that  reliable  and  maintainable  systems  can  be 
achieved  through  a  disciplined  engineering  approach  that  employs  the  best 
design  and  manufacturing  practices.  Emphasis  should  be  on  preventing  design 
deficiencies,  precluding  the  selection  of  unsuitable  parts  and  materials,  and 
minimizing  the  effects  of  variability  in  the  manufacturing  processes. 
Acquisition  officials  can  achieve  such  goals  by  using  lessons-learned  data  in 
their  decisionmaking  to  avoid  repeating  mistakes  and  profit  from  positive 
experiences. 


Objectives 


Our  audit  objectives  were  to  determine  whether  weapon  system  supportability 
problems  are  accurately  identified  and  reported  by  operational  commands  and 
whether  lessons-learned  data  were  adequately  considered  by  acquisition  officials 
during  weapon  system  development  or  modification.  We  also  reviewed  the 
applicable  internal  controls  related  to  this  process. 


Scope 


This  performance  audit  was  conducted  from  July  1992  through  March  1993  in 
accordance  with  auditing  standards  issued  by  the  Comptroller  General  of  the 
United  States,  as  implemented  by  the  Inspector  General,  DoD,  and  accordingly 
included  such  tests  of  internal  controls  as  were  deemed  necessary.  We 
evaluated  the  Military  Departments'  (MILDEPS)  policies  and  procedures  for 
reporting  weapon  system  supportability  problems.  We  also  assessed  the  use  of 
lessons-learned  data  by  acquisition  officials.  We  assessed  seven  major  weapon 
system  acquisition  programs  that  were  either  in  the  developmental  or 
modification  phase  of  the  acquisition  cycle  to  determine  whether  program 
officials  considered  and  used  lessons-learned  data  during  program 
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decisionmaking.  The  audit  included  a  review  of  documents  and  reports  dated 
from  May  1981  through  March  1993.  The  activities  visited  or  contacted  during 
the  audit  are  listed  in  Appendix  C. 


Internal  Controls 


In  evaluating  internal  controls,  we  reviewed  policies  and  procedures  used  by  the 
Military  Departments  for  using  and  disseminating  lessons-learned  data.  The 
audit  disclosed  internal  control  weaknesses  as  defined  by  Public  Law  97-255, 
Office  of  Management  and  Budget  Circular  A-123,  and  DoD  Directive  5010.38. 
The  Military  Departments  lessons-learned  data  bases  were  not  sufficiently 
reliable  to  allow  acquisition  officials  to  use  these  data  during  weapon  system 
development  and  modification.  Implementation  of  Recommendations  A.I., 
A. 3.,  and  A. 4.  will  correct  the  weaknesses  and  provide  assurance  that  lessons- 
learned  data  will  be  utilized  and  result  in  lower  weapon  system  life-cycle  costs. 
We  were  not  able  to  quantify  the  savings  related  to  cost  avoidance  since  future 
use  of  weapon  systems  could  not  be  projected.  Senior  officials  responsible  for 
internal  controls  in  the  areas  identified  will  be  provided  a  copy  this  report. 


Prior  Audits  and  Other  Reviews 


The  Department  of  Defense,  Office  of  the  Inspector  General;  Service  Audit 
Agencies;  and  the  General  Accounting  Office  have  not  specifically  assessed  the 
reporting  or  utilization  of  supportability-related  lessons-learned  data  within  the 
last  5  years. 
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Part  II  -  Findings  and  Recommendations 


Finding  A.  Using  Lessons-Learned  Data 

Program  offices  responsible  for  acquiring  major  weapon  systems  were 
not  adequately  evaluating  and  disseminating  lessons-learned  data  during 
weapon  systems  development  and  modification.  The  Army  and  Navy 
had  not  implemented  adequate  policies  for  identifying,  analyzing,  and 
disseminating  lessons-learned  data.  The  Air  Force  had  adequate  policies 
in  place;  however,  program  offices  were  not  adhering  to  these  policies. 
On  a  DoD-wide  basis,  there  was  insufficient  awareness  in  the  acquisition 
community  of  the  benefits  of  using  lessons-learned  data.  Consequently, 
acquisition  officials  usually  did  not  benefit  by  avoiding  past  mistakes  or 
profiting  from  positive  results  derived  from  data  collected  from  the 
extensive  reporting  systems  of  the  Military  Departments. 


Background 


Lessons-learned  data  are  recorded  experiences  of  value  to  the  conduct  of  current 
and  future  acquisition  programs.  They  are  conclusions  drawn  from  analysis  of 
feedback  on  deployed  systems  or  systems  currently  under  development  or 
modification.  Lessons  learned  may  be  positive  or  negative.  Positive  lessons 
show  successes  or  innovative  techniques,  while  negative  lessons  show 
deficiencies  or  problems  to  be  avoided.  There  are  two  categories  of  acquisition 
lessons  learned:  technical  and  management. 

Technical.  These  lessons  are  related  to  systems,  equipment,  and  components, 
including  hardware,  software,  support  equipment,  or  design  factors  that 
influence  the  performance  of  weapon  systems.  Most  lessons  learned  are 
technical  in  nature.  An  example  of  a  technical  lesson  learned  is  a  documented 
incident  that  involved  a  missile  component  that  malfunctioned  and  caused  the 
missile's  rocket  motor  to  ignite.  The  lesson-learned  data  showed  that  when 
certain  internal  components  of  a  missile  system  were  subjected  to  intense 
vibration,  the  components  became  loose  and  caused  missile  malfunctions.  A 
detailed  discussion  of  this  lesson  learned  is  in  Appendix  A. 

Management.  These  lessons  address  program  decisions  and  actions  in  areas 
such  as  budgeting,  contracting,  acquisition  planning,  and  data  management.  An 
example  of  a  management  lesson  learned  is  related  to  a  joint  Air  Force  and 
Navy  acquisition  program  that  involved  termination  of  an  internal  specification 
review  board  (SRB).  The  board  had  been  responsible  for  reviewing  system 
specification  documents.  Subsequently,  inconsistencies  among  the  specification 
documents  developed  for  which  the  Government  could  not  hold  the  contractor 
responsible.  The  lesson  learned  from  this  incident  was  that  using  SRBs  for  a 
joint  acquisition  program  will  minimize  inconsistencies  in  contractor 
specification  documents.  A  detailed  discussion  of  this  lesson  learned  is  in 
Appendix  A. 
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Assessing  Lessons-Learned  Programs  -  Army 


We  evaluated  acquisition  programs  for  three  major  weapon  systems  and  found 
that  lesson s-learned  data  bases  were  not  systematically  used  in  program 
decisionmaking.  The  three  acquisition  programs  assessed  were  M2/M3  Bradley 
Fighting  Vehicle,  Ml  Abrams  Tank,  and  Advanced  Anti-Tank  Weapon  System 
-  Medium  (Javelin). 

Bradley  and  Abrams  Programs.  Acquisition  officials  relied  on  contractors' 
analyses  and  the  corporate  knowledge  of  their  program  office  staff,  such  as 
engineers  and  logisticians,  for  lessons-learned  data.  Both  offices  also  required 
their  respective  contractors  to  acquire,  document,  and  store  lessons-learned  in 
data  bases  that  were  maintained  by  the  contractors;  however,  these  data  were 
not  available  for  sharing  with  other  Army  programs  or  the  other  Military 
Departments.  Furthermore,  these  data  are  not  transferable  to  the  Army's 
centralized  lessons-learned  data  base. 

Javelin  Program.  The  program  office  did  not  use  lessons-learned  data  from  or 
provide  input  to  the  Army's  lessons-learned  data  base  except  for  two  lesson- 
learned  reports  submitted  by  the  program  office  in  1990  and  1992.  These 
officials  relied  on  their  prime  contractors  and  the  corporate  knowledge  of 
program  office  staff  for  lessons-learned  information. 

Policies.  The  Army  Chief  of  Staff  in  1985  directed  the  establishment  of  a 
formal  system  for  capturing,  analyzing,  and  disseminating  lessons-learned 
information.  Several  regulations  result^  from  this  decision;  however,  these 
regulations  did  not  provide  adequate  guidance  for  all  aspects  of  acquisition 
lessons-learned  programs.  The  one  policy  that  did  apply  to  acquisition  lessons- 
learned  data  had  been  rescinded. 

o  Army  Regulation  11-33,  "Army  Lessons  Learned  Program:  System 
Development  and  Application,"  October  10,  1989,  established  the  Army's 
comprehensive  lessons-learned  system.  However,  this  regulation  focused 
primarily  on  tactical-type  lessons  learned. 

o  Army  Regulation  700-127,  "Integrated  Logistics  Support,"  July  17, 
1990,  requires  acquisition  officials  to  provide  input  on  initial  support  problems, 
support  innovations,  and  other  issues  to  the  Army's  Integrated  Logistics  Support 
(ILS)  lessons-learned  information  file.  This  regulation  focuses  primarily  on 
logistics-type  lessons-learned  data.  It  does  not  fully  address  acquisition-type 
lessons-learned  such  as  latent  design  defects. 

o  Army  Materiel  Command  Regulation  11-44,  "Army  Materiel 
Command  Lessons  Learned  Program,"  December  15,  1989,  (Rescinded) 
established  guidance  for  acquisition  officials  to  collect  operational,  technical, 
and  managerial  experiences  that  would  be  helpful  to  decisionmakers  and 
disseminate  these  lessons  learned  to  other  Army  activities.  This  regulation 
provided  adequate  guidance  for  acquisition  officials  to  use  lessons-learned  data 
during  their  decisionmaking  process;  however,  it  was  rescinded  in 
October  1992. 
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Lessons-Learned  Data  Base.  Responsibilities  for  maintaining  the  Army's 
lessons-leamed  data  base  were  delegated  to  the  Army's  Materiel  Readiness 
Support  Activity  (MRS  A).  Data  were  collected  by  MRS  A  on  major  training 
exercises,  combat  operations,  and  integrated  logistics  support  issues.  Data 
related  to  tactical-type  issues  were  computerized;  however,  data  related  to 
acquisition  and  logistics  issues  were  maintained  only  in  hard  copy. 
Consequently,  these  data  were  not  readily  accessable  to  program  officials. 


Assessing  Lessons-Learned  Programs  -  Navy 


Two  acquisition  programs  were  assessed;  F/A-18  Hornet  Aircraft,  managed  by 
the  Program  Executive  Officer  (PEO)  for  Tactical  Aircraft  Programs,  and  the 
Close-In  Weapon  System  (Phalanx)  (CIWS),  managed  by  the  Program 
Executive  Officer  for  Ship  Defense. 

F/A-18.  Although  the  Naval  Air  Systems  Command  has  had  a  centralized 
lessons-learned  data  base  since  1983,  there  was  no  evidence  that  F/A-18 
program  officials  used  the  data  base  in  their  decisionmaking  process.  These 
officials  used  internal  sources  they  believed  were  more  reliable  and  timely  than 
the  centralized  data  base.  Specifically,  they  relied  on  the  program  office  staff's 
corporate  knowledge  of  specific  weapon  system  problems  and  deficiencies  and 
analyses  compiled  by  prime  contractors. 

Close-In  Weapon  System  (Phalanx).  Neither  the  Naval  Sea  Systems 
Command  nor  the  program  office  had  a  requirement  for  acquisition  officials  to 
use  lessons-learned  data  in  their  decisionmaking  process.  The  program  office 
relied  on  the  contractor's  knowledge  and  program  office's  analyses  of  fleet 
reporting  on  equipment  failures  and  deficiencies. 

Policies.  There  was  no  Navy-wide  policy  requiring  acquisition  officials  to  use 
lessons-learned  data  in  their  decisionmaking.  Furthermore,  there  were  no 
requirements  within  either  the  PEO  or  program  management  organization  to  use 
lessons-learned  data.  However,  the  Naval  Air  Systems  Command  (NAVAIR) 
was  drafting  an  instruction  titled  "Naval  Aviation  Lessons  Learned  (NALL) 
Program,"  which  establishes  policies  and  procedures  and  assigns  responsibilities 
for  operating  the  NAVAIR  lessons-learned  program.  The  proposed  instruction 
would  require  all  NAVAIR  entities  to  use  and  participate  in  the  NALL 
program. 

Lessons-Learned  Data  Base.  The  NALL  program  was  established  in  1983  to 
support  the  Joint  Services  Advanced  Vertical  Lift  Aircraft  acquisition  program 
as  a  formalized  process  for  obtaining  lessons-learned  data  from  sources  such  as 
engineering  investigations,  inspection  survey  reports,  test  reports,  engineering 
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change  proposals,  and  maintenance  and  fleet  reporting.  However,  the  use  of 
this  data  base  by  acquisition  officials  was  optional  since  no  overall  Navy, 
NAVAIR,  or  PEG  policies  required  its  use. 


Assessing  Lessons-Learned  Programs  -  Air  Force 


Two  acquisition  programs  were  assessed:  F-16  Falcon  Aircraft,  managed  by 
the  Program  Executive  Officer  for  Tactical  and  Airlift  Programs  and  the 
Advanc^  Medium  Range  Air-to-Air  Missile  (AMR  A  AM),  manajged  by  the 
Program  Executive  Officer  for  Conventional  Strikes.  The  F-16  is  a  fielded 
system  undergoing  program  modifications.  The  AMRAAM  is  a  developmental 
program  in  acquisition  phase  III  with  a  Pre-planned  Product  Improvement 
Program  in  process.  Acquisition  officials  for  the  two  programs  did  not  use  the 
Air  Force's  formalized  lessons-leamed  data  base  in  their  decisionmaking 
process.  Program  officials  relied  on  corporate  knowledge  gained  from  their 
contractors  and  program  office  staff  for  lessons-learned  data. 

Policies.  The  Air  Force  had  formalized  policies  in  place  requiring  acquisition 
officials  to  use  lessons-leamed  data  in  their  decisionmaking  process. 
Specifically,  Air  Force  Logistics  Command  (AFLC)  and  Air  Force  Systems 
Command  (AFSC)  (now  combined  under  Air  Force  Materiel  Command) 
Regulation  800-37,  "Joint  AFLC/ AFSC  Lessons  Learned  Program,"  April  15, 
1988,  requires  program  offices  responsible  for  planning  and  acquiring  weapon 
systems  to  identify,  assess,  document,  and  disseminate  lessons-learned  data 
throughout  a  system's  life  cycle.  The  guidance  also  requires  program  offices  to 
make  maximum  use  of  lessons  learned  from  other  sources  to  avoid  repeating 
mistakes.  However,  the  Air  Force's  lessons-leamed  policy  was  not  adhered  to 
by  the  two  program  offices. 

Lessons-Learned  Data  Base.  The  Air  Force's  lessons-learned  data  base  was 
implemented  in  1977  and  was  fully  automated  in  1978.  The  data  base  basically 
contains  management  and  technic^ -type  lessons-learned  information.  The  data 
base  also  serves  the  other  two  MILDEPS.  Although  formalized  procedures 
exist  for  using  the  data  base,  use  of  the  data  base  by  acquisition  officials  was 
almost  nonexistent  because  of  limited  feedback  from  the  acquisition  community. 
As  a  result,  participation  in  the  Air  Force's  lessons-learned  program  by  the 
two  acquisition  programs  reviewed  was  limited. 
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Consequences  and  Benefits 


The  failure  of  acquisition  officials  to  develop  effective  procedures  or  adhere  to 
established  policies  for  using  lessons-learned  data  during  the  acquisition  process 
resulted  in  missed  opportunities  for  acquisition  programs  to  avoid  repeating 
mistakes  or  gain  by  incorporating  successful  results.  We  found  several 
examples  that  show  the  consequences  of  program  offices  not  participating  in  an 
exchange  of  lessons-learned  information.  We  also  show  an  example  of  benefits 
gained  from  lessons-learned  information. 

Consequences.  To  prevent  a  slippage  in  the  AMR  A  AM's  initial  fielding 
schedule,  acquisition  officials  decided  to  begin  low-rate  initial  production  with 
only  a  preliminary  missile  design.  At  that  time,  the  missile  design  was  not  fully 
capable  against  more  sophisticated  enemy  electronic  countermeasures. 
Realizing  this  limitation,  acquisition  officials  pursued  and  obtained  approval  to 
proceed  with  low-rate  initial  production.  These  officials  ignored  experience  that 
shows  moving  an  acquisition  program  prematurely  into  the  production  phase 
with  a  limited  system  design  can  result  in  significant  problems. 

Currently,  the  AMRAAM  program,  which  is  a  joint  Air  Force  and  Navy 
developmental  effort,  is  confronted  with  a  possible  slip  in  the  Navy's  initial 
operating  capability  date  by  as  much  as  a  year  to  allow  the  Navy  to  implement 
and  complete  an  engineering  change  program  for  resolving  interface  deficiencies 
between  the  aircraft  and  the  missile. 

Lessons  Exchanged.  This  example  involved  the  Army  Maneuver  Control 
System  (MCS).  Lessons-learned  data  that  had  been  analyzed  and  compiled  by 
the  Army's  Combined  Arms  Center  (CAC)  were  not  shared  with  potential 
users.  Although  these  data,  which  had  been  compiled  frorn  the  Operations 
Desert  Shield  and  Desert  Storm,  represented  valuable  information  to  acquisition 
officials,  CAC  did  not  disseminate  this  report  to  the  MCS  program  office. 
Therefore,  specific  deficiencies  in  training  and  use  of  the  MCS  system  were  not 
available  to  the  program  office. 

Benefits  Realized.  When  properly  used,  lessons-learned  data  can  provide 
valuable  opportunities  for  acquisition  officials  to  avoid  repeating  costly 
mistakes.  For  example,  lessons-learned  data  obtained  from  the  Army's  Javelin 
program  have  allowed  other  DoD  acquisition  programs  to  benefit  from  research 
efforts  related  to  the  Javelin's  state-of-the-art  development  on  seeker  research. 

In  October  1992,  the  Under  Secretary  of  Defense  for  Acquisition  designated  the 
Javelin  as  the  lead  acquisition  program  in  the  development  of  second  generation 
imaging  device,  which  is  a  component  of  the  system's  guidance  section.  The 
Javelin  had  encountered  significant  problems  that  resulted  in  major  program  cost 
increases  and  schedule  slippage.  Problems  associated  with  the  development  of 
second  generation  imaging  technology  were  related  to  the  capability  to  produce 
enough  quality  devices.  Since  this  device  was  to  be  common  among  other  DoD 
programs,  lessons-learned  data  from  the  Javelin  research  efforts  were  shared 
with  the  other  Military  Departments  and  should  preclude  them  from 
encountering  the  same  developmental  problems  with  their  imaging  devices. 
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Lessons-Learned  Committee.  The  Navy  Antiair  Warfare  Weapon  System 
Lessons-Learned  Committee  represented  one  of  the  most  formalized  processes 
used  by  Navy  acquisition  officials  for  capturing  and  using  lesson s-learned  data. 
The  committee  was  established  in  1984  to  identify  problems  and  define  solutions 
for  certain  classes  of  guided  missile  cruisers  and  destroyers  as  they  entered  the 
fleet.  The  committee  used  fleet  reporting  via  the  Navy's  Maintenance  and 
Material  Management  System,  which  documents  equipment  maintenance-related 
problems.  This  reporting  is  captured  and  analyzed  by  contractors  for  potential 
lessons  learned.  The  results  are  provided  to  the  Antiair  Warfare  Weapon 
System  Lessons-Learned  Committee  for  action. 


Reporting  Systems 


The  Military  Departments  used  various  methods,  such  as  maintenance  and 
material  reporting  systems,  quality  deficiency  reporting,  and  engineering 
investigations  to  report  supportability  problems  relative  to  equipment  failures 
and  deficiencies.  Our  review  showM  these  reporting  systems  were  collecting 
extensive  information  on  various  types  of  incidents.  Although  available  to 
acquisition  program  offices  to  evaluate,  the  lessons-learned  data  were  seldom 
used  by  anyone  outside  the  logistics  community.  We  believe  the  significant 
resources  expended  by  the  Military  Departments  to  manage  these  reporting 
systems  can  be  further  enhanced  if  used  by  acquisition  program  offices.  Other 
than  the  Antiair  Warfare  Weapon  System  Lessons-Learned  Committee,  as 
discussed  above,  we  did  not  find  program  offices  using  these  systems  as  a 
lessons-learned  source. 


Conclusion 


Utilization  of  lessons-learned  data  in  the  decisionmaking  process  during  the 
development  or  modification  of  weapon  systems  can  result  in  reduced  life-cycle 
costs  and  improved  readiness.  Examples  show  when  lessons-learned  data  are 
shared  among  the  Military  Departments  and  used  by  acquisition  officials, 
mistakes  are  not  repeated.  The  Military  Departments  had  some  aspects  of 
lessons-learned  policies  in  place.  However,  except  in  the  Air  Force,  the 
policies  were  generally  inadequate  and  not  sufficiently  comprehensive  to  be 
effective. 
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Recommendations,  Management  Comments,  and  Audit 
Response 


1.  We  recommend  that  the  Assistant  Secretary  of  the  Army  (Research, 
Development  and  Acquisition); 

a.  Modify  existing  policies  to  require  the  inclusion  of  acquisition- 
type  lessons-learned  data  in  the  Army  data  base  maintained  by  the  Materiel 
Readiness  Support  Activity. 

Army  Response.  The  Office  of  the  Assistant  Secretary  of  the  Army  (Research, 
Development  and  Acquisition)  agreed  with  the  recommendation  and  stated  that 
the  Army  would  provide  an  acquisition  lessons-learned  data  base  to  be 
maintained  by  the  Materiel  Readiness  Support  Activity.  Guidance  would  be 
published  in  the  fourth  quarter  of  FY  1994.  The  full  text  of  Army's  comments 
is  in  Part  IV  of  this  report. 

Audit  Response.  We  consider  the  Army's  cornments  to  be  responsive  and  no 
further  comments  are  required  for  Recommendation  l.a. 

b.  Require  program  offices  to  use  the  Army's  formalized  lessons- 
learned  data  during  the  development  or  modification  of  major  weapon 
systems. 

Army  Response.  Management  agreed  with  the  recommendation  and  stated  that 
the  Army  is  broadening  the  scope  of  data  collection.  Technical  data  will  be 
maintained  by  the  Army  Materiel  Command  but  will  be  processed  through  and 
shared  by  the  Center  for  Army  Lessons-Learned. 

Audit  Response.  Although  management's  response  is  positive,  it  does  not 
specifically  address  the  issue  of  requiring  program  offices  to  use  lessons-learned 
data.  When  gathered,  analyzed,  and  applied,  lessons  learned  can  reduce 
program  risk.  We  request  the  Army  provide  a  firmer  commitment  to  use 
lessons  learned  once  the  data  base  has  been  established. 

2.  We  recommend  that  the  Army  Deputy  Chief  of  Staff  for  Logistics 
automate  the  Army's  lessons-learned  data  into  an  easy-to-retrieve  data 
base. 

Army  Response.  The  Army  is  automating  Integrated  Logistics  Support 
Lessons  Learned  into  a  simplified  system  so  retrieval  will  be  easier.  The 
vehicle  for  this  automation  is  the  Logistics  Planning  and  Requirements 
Simplification  System.  ILS  Lessons  Learned  will  be  incorporated  into  the 
system's  electronic  bulletin  board,  and  further  automation  will  take  place  as 
priority  or  resources  permit.  The  full  text  of  Army's  comments  is  in  Part  IV  of 
this  report. 

Audit  Response.  We  consider  the  Army's  comments  to  be  responsive  and  no 
forther  comments  are  required  for  Recommendation  2. 
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3.  We  reconunend  that  the  Assistant  Secretary  of  the  Navy  (Research, 
Development  and  Acquisition)  revise  existing  policies  to  require  program 
offices  to  access  lessons-learned  data  bases  during  vi^eapon  system 
development  and  modifications. 

Navy  Response.  The  Assistant  Secretary  of  the  Navy  (Research,  Development 
and  Acquisition)  partially  concurred.  The  Navy  stated  that  DoD  Instruction 
5000.2  specifically  requires  review  of  lessons-learned  data  for  many  functional 
areas,  as  discussed  in  Parts  6,  7,  and  10.  Management  stated  that  DoD 
Directive  5000.1  can  not  be  supplemented  without  approval  by  DoD  or  by 
statute  and  therefore  recommended  that  "other  'lessons  learned'  areas"  be 
considered  by  the  Under  Secretary  of  Defense  (Acquisition)  for  later  revisions 
to  DoD  Instruction  5000.2.  Notwithstanding,  Navy  will  advise  program 
managers  of  the  availability  of  operational  data  from  Navy  reporting  systems 
such  as  the  Maintenance  and  Material  Management  System.  The  full  text  of 
Navy  comments  is  in  Part  IV. 

Audit  Response.  The  audit  report  discusses  DoD  Instruction  5000.2  with 
specific  reference  to  lessons  learned  (see  Background  section).  Management- 
cited  references  to  the  Instruction  do  not  discuss  lessons  learned.  Also,  we  are 
aware  of  the  policy  regarding  supplementation  to  the  Instruction.  A 
memorandum  advising  program  managers  of  the  availability  of  operational  data 
through  Navy  reporting  systems  such  as  the  Maintenance  and  Material 
Management  System  will  not  meet  the  intent  of  the  recommendation.  However, 
operational  lessons  learned  are  not  the  subject  of  this  audit;  moreover,  Chief  of 
Naval  Operation  Instruction  3500.37,  "Navy  Lessons  Learned  System  (NLLS)," 
March  17,  1992,  already  provides  for  the  submission,  validation,  and 
distribution  of  maritime-specific  lessons  of  an  operational  nature,  which  would 
provide  more  refined  information  than  reporting  systems  such  as  Maintenance 
and  Material  Management.  We  believe  that  the  Navy  should  issue  policy 
specifically  addressing  acquisition  lessons  learned. 

4.  We  recommend  that  the  Commander,  Air  Force  Materiel  Command, 
require  program  offices  to  use  and  disseminate  lessons-learned  data  during 
the  development  and  modification  of  major  weapon  systems. 

Air  Force  Response.  The  Assistant  Secretary  of  the  Air  Force  (Acquisition), 
and  Deputy  Assistant  Secretary  (Management  Policy  and  Program  Integration) 
concurred.  Management  stated  that  the  Air  Force  was  instituting  specific 
processes  for  each  major  system  acquisition  area.  According  to  the  approved 
Air  Force  supplement  to  DoD  Instruction  5000.2,  all  Air  Force  functional 
activities  will  provide  appropriate  lessons  for  the  data  base.  Each  program 
office  will  periodically  review  the  appropriate  lessons  learned  and  apply  the 
applicable  best  practices.  The  full  text  of  the  comments  is  in  Part  IV  of  this 
report. 

Audit  Response.  We  consider  the  Air  Force  comments  to  be  responsive  to  the 
recommendation;  no  further  comments  are  required. 
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Finding  B.  Joint-Service  Lessons- 
Learned  Program 

The  Joint  Logistics  Commanders'  (JLC)  attempt  to  establish  an 
integrated  lesson s-learned  exchange  program  among  the  Military 
Departments  has  languished  since  the  program's  inception  in  1989.  The 
JLC's  Memorandum  of  Agreement  (MOA),  which  authorized  the 
program,  was  vague  and  limited  primarily  to  logistics-type  data,  thus 
excluding  acquisition-related  lesson s-leamed  data.  In  addition,  the 
agreement  assigned  program  implementation  responsibilities  to 
organizational  entities  that  lacked  authority  to  initiate  significant  program 
actions.  As  a  result,  the  benefits  that  could  be  derived  from  such  a 
program  have  not  been  realized. 


Background 


Memorandum  of  Agreement.  On  March  16,  1989,  the  JLC  signed  an  MOA 
to  establish  a  system  for  exchanging  lesson  s-learned  data  among  the  Military 
Departments.  The  agreement  between  the  Army  Materiel  Command,  the  Office 
of  the  Chief  of  Naval  Operations,  and  the  Air  Force  Logistics  Command  and 
the  Air  Force  Systems  Command  (combined  as  the  Air  Force  Materiel 
Command  in  July  1992),  outlined  the  objectives  regarding  the  exchange  of 
logistics  lessons-learned  data.  The  MOA  stipulated  four  specific  objectives: 

o  Exchange  lessons-learned  and  supporting  data  freely  among  all 
Military  Departments. 

o  Provide  for  timely  exchange  of  lessons-learned  data  while  minimizing 
administration  and  coordination  required  to  effect  the  exchange. 

o  Use  a  minimum  standard  format  for  documenting  individual  lessons- 
learned  within  and  among  the  Military  Departments. 

o  Retain  independence  of  each  Military  Department's  lessons-learned 
data  base  since  each  Military  Department's  lessons-learned  data  are  essentially 
hardware-specific. 

Ad  Hoc  Working  Group.  To  fulfill  the  MOA  objectives,  an  Ad  Hoc  working 
group  was  formed  consisting  of  representatives  from  each  Military  Department. 
The  MOA  called  for  the  Ad  Hoc  group  to  meet  at  least  annually  to  manage  the 
JLC  program.  The  group  has  met  annually  since  1989  to  discuss  problems  and 
other  issues  associated  with  the  Military  Departments'  attempt  to  accomplish  the 
MOA  objectives.  The  group  also  considered  alternative  strategies. 
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Program  Progress 


Our  analysis  of  program  minutes  and  discussions  with  members  of  the  Ad  Hoc 
group  disclosed  that  the  program  has  made  only  minimal  progress  since  its 
inception.  Our  review  of  the  group's  minutes  disclosed  that  after  almost  4  years 
of  planning,  the  group  is  not  close  to  fulfilling  the  MOA's  objectives. 
Specifically,  the  minutes  disclosed  a  lack  of  significant  progress  in  establishing 
a  mechanism  for  exchanging  aviation  lessons-learned  data  among  the  Military 
Departments.  Although  the  Navy  and  Air  Force  have  made  some  progress  in 
formulating  plans  for  exchanging  aviation  lessons-learned  data,  the  Army  has 
not  demonstrated  much  progress  or  shown  much  interest  in  participating  in  the 
effort.  The  group  had  recognized  earlier  that  the  Army's  lessons-learned  data 
base,  which  was  in  hard  copy  only,  was  not  compatible  with  the  Navy  and  Air 
Force's  computerized  systems. 

At  the  Ad  Hoc  group's  annual  meeting  in  1990,  the  Army  considered  the 
possibility  of  using  lessons-learned  data  stored  at  the  Defense  Logistics  Studies 
Information  Exchange  (DLSIE)  to  establish  compatibility  with  the  Na.vy  and 
Air  Force.  However,  the  issue  was  dropped  after  an  Army  internal 
reorganization  resulted  in  DLSIE  being  transferred  from  the  Army  Materiel 
Command.  There  is  no  evidence  that  the  Army  considered  any  other 
alternative. 


Agreement  Language 


Concerns  have  been  raised  within  the  Ad  Hoc  group  over  the  MOA's  language. 
For  example,  a  briefing  paper  on  the  feasibility  of  integrating  data  bases  stated 
that  the  language  of  the  MOA  was  too  general  to  effectively  standardize  the 
Military  Departments'  lessons-learned  process.  The  paper  concluded  that  each 
Military  Department  has  a  distinct  set  of  procedures  for  the  retrieval,  validation, 
and  maintenance  of  its  respective  data  base,  leading  to  a  lack  of  standardization 
in  procedures  among  the  Military  Departments. 

Currently,  the  group  is  attempting  to  address  the  standardization  issue.  Also, 
one  Military  Department  representative  proposed  that  an  executive  agency  be 
designated  to  establish  direction  and  define  specific  milestones  for  completing 
group  taskings. 


Ad  Hoc  Group  Representation 


Since  its  inception,  the  Ad  Hoc  group  has  been  comprised  solely  of  logisticians 
from  organizations  within  the  Military  Departments  who  did  not  have  authority 
to  implement  significant  program  actions.  In  addition,  there  was  no 
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representation  from  acquisition  organizations.  Furthermore,  the  minutes  of  the 
annual  meetings  did  not  reflect  coordination  by  the  logisticians  with  acquisition 
officials. 

As  shown  in  Finding  A  of  this  report,  acquisition  managers  responsible  for 
major  weapon  system  development  do  not  normally  query  their  Military 
Departments’  formal  lessons-learned  data  bases  before  major  program  decisions. 
We  believe  that  without  active  participation  by  the  Military  Departments' 
acquisition  organizations  in  the  JLC  planning  process,  it  is  highly  unlikely  that 
acquisition  officials  will  effectively  use  the  exchange  program  in  their 
decisionmaking. 


Conclusion 


Although  nearly  4  years  of  planning  have  transpired  since  the  JLC  signed  the 
MOA  for  establishing  an  integrated  lessons-learned  program,  the  Ad  Hoc  group 
has  made  little  progress  in  developing  an  effective  plan  for  its  implementation. 
These  4  years  have  been  marked  by  a  lack  of  measurable  progress  and  a  lack  of 
defined  milestones  for  completing  taskings. 


Recommendations  for  Corrective  Action  and  Management 
Response 


We  recommend  that  the  Joint  Logistics  Commanders: 

1.  Designate  an  executive  agency  to  be  responsible  for  program  oversight 
and  require  the  inclusion  of  the  acquisition  community  representation  in 
the  Ad  Hoc  group. 

2.  Require  the  executive  agency  to  establish  more  specific  guidelines  for 
defining  program  taskings  and  milestone  dates. 

Joint  Logistics  Commanders  Response.  The  JLC  did  not  provide  comments 
within  the  specified  time.  The  official  who  is  serving  as  the  focal  point  for 
coordination  of  the  JLC  response  indicated,  however,  that  he  would  initiate 
action  so  we  will  receive  a  response  to  the  final  report. 
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Appendix  A.  Lessons-Learned  Examples 


Synopsized  below  are  lessons-learned  examples  from  the  Military  Departments' 
lesson-learned  data  bases,  as  discussed  in  Finding  A  of  this  report. 


Technical  Lesson-Learned  Example 


Problem:  The  metal  end  plug  of  an  AMRAAM  rocket  motor  igniter  tube 
disengaged  and  became  lodged  in  the  fuel  tank.  The  igniter  produced  localized 
heating  during  vibration  testing  that  caused  the  rocket  motor  to  ignite. 

Discussion:  During  developmental  vibration  testing,  the  threaded  end  plug  of 
the  AMRAAM  rocket  motor  igniter  disengaged  and  fell  into  a  radial  slot  in  the 
fuel  tank.  Since  the  designs  of  the  AMRAAM  motor  igniter  and  the  fuel  tank 
are  representative  of  tactical  missile  rocket  motor  designs,  this  problem  could  be 
encountered  during  the  design  of  future  tactical  rocket  motors. 

Lesson-Learned  Data:  Components  internal  to  solid  propellant  rocket  motors 
that  are  subject  to  intense  vibration  may  become  loose  and  fall  out.  This 
situation  could  cause  the  missile's  rocket  motor  to  ignite. 


Management  Lesson-Learned  Example 


Problem:  The  Joint  Air  Force  and  Navy  acquisition  program,  AMRAAM, 
used  an  internal  SRB  for  many  years.  Use  of  the  SRB  resulted  in  consistency  in 
system  specification  documents  that  were  produced  by  contractors.  However, 
after  SRBs  were  discontinued,  inconsistencies  in  specification  documents  began 
to  develop. 

Discussion:  The  Military  Departments  have  varying  requirements  for  content, 
format,  and  procedures  used  in  the  preparation  of  system  specification 
documents.  The  lack  of  effective  control  over  the  development  of  these 
documents  in  joint  programs  results  in  inconsistencies  in  their  preparation. 
Without  SRBs  to  monitor  document  preparation,  contractors  are  in  control  of 
interpreting  and  pricing  specification  requirements,  generally  to  the 
Government's  disadvantage. 

Lesson-Learned  Data:  Inconsistencies  in  Government-prepared  specification 
documents  create  interpretation  problems  for  the  contractor.  The  use  of  SRBs 
in  joint  acquisition  programs  can  alleviate  this  problem. 
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Resulting  From  Audit 


Recommendation 

Reference 

Description  of  Benefit 

Amount  and/or 

Type  of  Benefit 

A.l.a. 

Economy  and  Efficiency  and 

Internal  Control.  Will  ensure 
adequate  data  for  future  program 
decisions. 

Undeterminable 
Monetary  Benefits. 
Unable  to  project 
future  use. 

A.l.b. 

Economy  and  Efficiency  and 

Internal  Control.  Will  ensure 
adequate  data  for  future  program 
decisions. 

Undeterminable 
Monetary  Benefits. 
Unable  to  project 
future  use. 

A.2. 

Economy  and  Efficiency.  Will 
allow  for  more  effective  use  of 
lessons-learned  data  base. 

Undeterminable 
Monetary  Benefits. 
Based  on  future  use  of 
system. 

A.3. 

Economy  and  Efficiency  and 

Internal  Controls.  Will  allow  for 
more  effective  use  of  lessons- 
learned  data  base. 

Undeterminable 
Monetary  Benefits. 
Unable  to  project 
future  use. 

A.4. 

Economy  and  Efficiency  and 

Internal  Controls.  Will  ensure 
adequate  data  for  future  program 
decisions. 

Undeterminable 
Monetary  Benefits. 
Based  on  future  use  of 
program. 

B.l. 

Program  Results.  Will  result  in 
more  effective  use  of  Ad  Hoc 
group's  resources. 

Nonmonetary. 

B.2. 

Program  Results.  Will  ensure 

JLC's  goals  are  achieved. 

Nonmonetary. 
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Appendix  C.  Activities  Visited  or  Contacted 


Office  of  the  Secretary  of  Defense 

Office  of  the  Assistant  Secretary  of  Defense  (Production  and  Logistics),  Washington, 
DC 


Department  of  the  Army 

Deputy  Chief  of  Staff  for  Logistics,  Washington,  DC 
Deputy  Chief  of  Staff  for  Operations,  Washington,  DC 
U.S.  Army  Materiel  Command,  Alexandria,  VA 

U.S.  Army  Materiel  and  Readiness  Support  Activity,  Lexington,  KY 
U.S.  Army  Materiel  Systems  Analysis  Activity,  Aberdeen  Proving  Grounds,  MD 
U.S.  Army  Missile  Command,  Redstone  Arsenal,  AL 
U.S.  Army  Tank-Automotive  Command,  Warren,  MI 
Forces  Command,  Ft.  McPherson,  GA 
1st  Cavalry  Division,  Ft.  Hood,  TX 

Headquarters,  U.S.  Army  Pacific  Command,  Ft.  Shafter,  HI 
H  Company,  25th  Aviation  Regiment,  Schofield  Barracks,  HI 
Combat  Development  Center,  Ft.  Benning,  GA 
Combined  Arms  Center,  Ft.  Leavenworth,  KS 
National  Training  Center,  Ft.  Irwin,  CA 

U.S.  Army  Night  Vision  and  Electro  Optics  Laboratory,  Ft.  Belvoir,  VA 


Department  of  the  Navy 

Office  of  the  Assistant  Secretary  of  the  Navy  for  Research,  Development  and 
Acquisition,  Washington,  DC 
Naval  Air  Systems  Command,  Washington,  DC 
Naval  Air  Warfare  Center,  Patuxent  River,  MD 
Naval  Sea  Systems  Command,  Washington,  DC 

Ship  Parts  Control  Center  and  Naval  Sea  Logistics  Center,  Mechanicsburg,  PA 
Naval  Supply  Systems  Command,  Washington,  DC 
Commander  in  Chief  Atlantic,  Norfolk,  VA 
Commander  in  Chief  Pacific,  Honolulu,  HI 
Fleet  Marine  Force  Pacific,  Honolulu,  HI 
Marine  Corps  Combat  Development  Center,  Quantico,  VA 


Department  of  the  Air  Force 

Deputy  Chief  of  Staff  for  Logistics,  Washington,  DC 

Headquarters,  Air  Force  Materiel  Command,  Wright-Patterson  Air  Force  Base,  OH 
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Appendix  C.  Activities  Visited  or  Contacted 


Air  Logistics  Center,  McClellan  Air  Force  Base,  CA 
Air  Logistics  Center,  Robins  Air  Force  Base,  GA 
9th  Air  Base  Wing,  Logistics  Group,  Beale  Air  Force  Base,  CA 
Aeronautical  Systems  Command,  Eglin  Air  Force  Base,  FL 
Headquarters,  Air  Combat  Command,  Langley  Air  Force  Base,  VA 
Headquarters,  Pacific  Command  Air  Force,  Hickam  Air  Force  Base,  HI 
Headquarters,  Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  Air  Force 

Base,  NM  . 

Air  Force  Operational  Test  and  Evaluation  Activity,  Eglin  Air  Force  Base,  FL 


Non-Government  Activities 

General  Dynamics  Land  Systems  Division,  Warren,  MI 
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Appendix  D.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  (Acquisition) 
Director,  Joint  Staff 


Department  of  the  Army 

Assistant  Secretary  of  the  Army  (Research,  Development  and  Acquisition) 
Inspector  General,  Department  of  the  Army 
U.S.  Army  Deputy  Chief  of  Staff  (Logistics) 

Commanding  General,  U.S.  Army  Materiel  Command 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management) 

Assistant  Secretary  of  the  Navy  (Research,  Development  and  Acquisition) 

Deputy  Chief  of  Naval  Operations  (Logistics) 

Commander,  U.S.  Marine  Corps  Research,  Development  and  Acquisition  Command 
U.S.  Marine  Corps  Deputy  Chief  of  Staff  (Installations  and  Logistics) 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Acquisition) 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
U.S.  Air  Force  Deputy  Chief  of  Staff  (Logistics) 

Commander,  Air  Force  Materiel  Command 


Defense  Agencies 

Director,  Defense  Logistics  Agency 


Non-DoD  Organizations 

Office  of  Management  and  Budget 

U.S.  General  Accounting  Office,  National  Security  and  International  Affairs  Division, 
Technical  Information  Center 


22 


Appendix  D.  Report  Distribution 


Chairman  and  Ranking  Minority  Member  of  the  following  Congressional  Committees 
and  Subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 
House  Committee  on  Armed  Services 
House  Committee  on  Government  Operations 

House  Subcommittee  on  Legislation  and  National  Security,  Committee  on 
Government  Operations 
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Part  IV  -  Management  Comments 
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Department  of  the  Army  Comments 


DEPARTMENT  OF  THE  ARMY 
OFFICE  OF  THE  ASSISTANT  SECRETARY 
RESEARCH  DEVELOPMENT  AND  ACQUISITION 
103  ARMY  PENTAGON 
WASHINGTON  DC  20310  0103 


0  I  SEP  1993 


SARD-RP 


MEMORANDUM  FOR  OFFICE  OF  THE  INSPECTOR  GENERAL 

(AUDITING)  DEPARTMENT  OF  DEFENSE 

SUBJECT;  Draft  Report  on  Lessons  Learned  Programs 

Within  the  Military  Departments  (Project  No. 
2AG-50007) 


In  response  to  your  memorandum  subject  above,  dated 
June  24,  1993  the  Army  submits  the  following  response: 

Recommendation  la.  Modify  existing  policies  to 
require  the  inclusion  of  acquisition-type  lessons- 
learned  data  in  the  Army  data  base  maintained  by  the 
Materiel  Readiness  Support  Activity. 

We  agree  with  the  recommendation  contained  in  the 
report.  We  are  developing  a  DA  Pamphlet  that  covers  the 
acquisition  process.  This  new  pamphlet  will  include 
provisions  for  an  acquisition  lessons-learned  data  base 
maintained  by  the  Materiel  Readiness  Support  Activity. 

We  should  publish  the  pamphlet  in  the  4th  quarter  of  FY 
1994. 

Recommendation  lb.  Require  program  offices  to  use 
the  Army's  formalized  lessons-learned  data  during  the 
development  or  modification  of  major  weapon  systems. 

We  also  agree  with  this  recommendation.  The  Army's 
Training  and  Doctrine  Command  is  the  Army's  executive 
agent  for  lessons-learned  and  directs  the  Center  for  the 
Army  Lessons-Learned  Program.  Although  there  is  no 
requirement  in  the  lessons-learned  program  for  acquisi¬ 
tion  managers,  the  Army  is  broadening  the  Center's 
collection  teams.  Technical  data  will  be  maintained  by 
the  Army  Materiel  Command  but  will  be  processed  through 
and  shared  by  the  Center  for  Army  Lessons-Learned . 

Recommendation  2.  We  recommend  that  the  Army 
Deputy  Chief  of  Staff  for  Logistics  automate  the  Army's 
lessons-learned  data  into  an  easy-to-retrieve  data  base. 


Prlntttf  on  Ricyctad  Pap«r 
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Department  of  the  Army  Comments 


-2- 

Integrated  Logistics  Support  (ILS)  Lessons-Learned 
provides  the  knowledge  base  for  the  Logistics  Planning 
and  Requirements  Simplification  System  (LOGFARS).  This 
is  an  ILS  expert  system  based  on  artificial  intelli¬ 
gence.  ILS  Lessons-Learned  will  be  incorporated  into 
the  LOGPARS  electronic  bulletin  board  providing  access 
to  Lessons-Learned.  Other  automation  Lessons-Learned 
will  occur  as  priority  or  resources  permit, 

POC  for  this  action  is  Mr.  R.  Finnigan,  DSN 
225-0506. 


GEORGE  J.  SAVITSKE 
Colonel,  GS 
Director,  Acquisition 

and  Industrial  Base  Policy 
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Department  of  the  Navy  Comments 


DEPARTMENT  OF  THE  NAVY 

OFFICE  OF  THE  ASSISTANT  SECRETARY 
(Research.  Development  and  Acquisition) 
WASHINGTON,  D  C  20350-1000 

4  3  AUG  1893 


MEMORANDUM  FOR  THE  DEPARTMENT  OF  DEFENSE  INSPECTOR  GENERAL 


Subj:  DRAFT  REPORT  ON  LESSONS-LEARNED  PROGRAMS  WITHIN  THE 

MILITARY  DEPARTMENTS  (PROJECT  NO.  2AG-5007) 

Ref:  (a)  DODIG  Memo  of  24  June  1993 

End:  (1)  DON  Response  to  Draft  Audit  Report 

I  am  responding  to  the  draft  audit  report  forwarded  by 
reference  (a)  concerning  a  lessons-learned  program  within  the 
Department  of  the  Navy. 

The  Department  of  Navy  response  is  provided  at  enclosure 
(1) .  We  generally  agree  with  the  draft  report  finding  but 
believe  that,  if  DODIG  desires  that  the  subject  "lessons  learned” 
specifically  be  addressed  as  a  matter  of  "policy”  to  program 
managers,  that  it  be  incorporated  into  the  DOD  5000  series 
instruction  and  manual.  Notwithstanding,  we  will  initiate  a  memo 
to  program  managers  advising  them  of  the  availability  of  a 
variety  of  operational  "lessons  learned"  data  bases  to  assist 
them  in  the  acquisition  process  and  encourage  their  use. 

SULC.WtH _ 

Edward  C.  Whitman 


Copy  to: 
NAVINSGEN 
NAVCOMPT  (NCB-53) 
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Department  of  the  Navy  Comments 


Department  of  the  Navy  Response 
to 

DODIG  Draft  Report  of  June  24,  1993 
on 

Lessons-Learned  Programs  Within  the  Military  Departments 
Project  No.  2AG-5007 


Finding  A: 


Program  offices  responsible  for  acquiring  major  weapons  systems 
were  not  adequately  evaluating  and  disseminating  lessons-learned 
data  during  weapon  systems  development  and  modification.  The  Army 
and  Navy  had  not  Implemented  adequate  policies  for  identifying, 
analyzing,  and  disseminating  lessons-learned  data.  The  Air  Force 
had  adequate  policies  in  place;  however,  program  offices  were  not 
adhering  to  these  policies.  Consequently,  acquisition  officials 
did  not  benefit  by  avoiding  past  mistakes  or  profiting  from 
positive  results  derived  from  data  collected  from  extensive 
reporting  systems  of  the  Services. 

Recommendation  A-3; 

We  recommend  that  the  Assistant  Secretary  of  the  Navy  (Research, 
Development,  and  Acquisition)  revise  existing  policies  to  require 
program  offices  to  access  lessons-learned  data  bases  during  weapon 
system  development  and  modifications. 

DON  Position: 

Partially  concur,  DOD  Instruction  5000,2  already  requires  the  use 
of  lessons  learned  data.  Specifically,  it  requires  review  of 
lessons  learned,  prior  deficiencies  and  problem  areas,  and  hazards 
in  many  functional  areas,  such  as: 

•  Maintainability  analyses  (Part  6  pages  6-C-4) 

•  Software  error  data  collection  (6-D-1-2) 

•  Human  factors  (6-H-3) , 

•  System  safety,  health  hazard,  and  environmental  effects 
(6-1-6) 

•  Government-Industry  Data  Exchange  (GIDEP)  (6-N-2) 

•  Risk  templates-major  risk  areas  common  to  defense  programs 
(6-0-3) 

•  Contractor  quality  data  (6-P-2) 

•  Quality  deficiency  reporting  (6-P-3) 

•  Support  cost  drivers  (Part  7  pages  7-A-2-2) 

•  Manpower  impact  compared  to  predecessor  system (s) ,  safety  and 
health  hazards,  and  current  human  system  cost  drivers  (7-B-3) 
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Department  of  the  Navy  Comments 


t  Offeror'S  recent  and  past  history  relative  to  such  indicators 
as  quality,  timeliness,  cost,  schedule,  operational 
effectiveness  and  suitability  (Part  10  page  lo-B-6) 

The  DOD  Directive  5000.1  specifically  directs  that  the 
requirements  therein  shall  not  be  supplemented  except  as  prescribed 
by  statute  or  with  prior  approval  of  DOD  (Cover  memo  of  DODD  5000.1 
dated  23  Feb  91,  para.  G)  .  Thus,  if  other  "lessons  learned"  areas 
need  to  be  reviewed  by  the  program  offices  as  a  matter  of  policy, 
recommend  these  requirements  be  identified  by  USD(A&T)  for 
incorporation  in  subsequent  revisions  to  DODI  5000.2. 

In  the  meantime,  Navy  will  initiate  a  memo  to  the  program 
managers  advising  them  of  the  availability  of  operational  "lessons 
learned"  data  bases,  such  as,  3M,  quality  deficiency  reporting, 
engineering  investigations,  etc.,  to  assist  them  in  the  acquisition 
process  and  encourage  their  use.  This  memo  will  be  issued  by  30 
September  1993. 
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Department  of  the  Air  Force  Comments 


OFFICE  OF  THE  ASSISTANT  SECRETARY 


MEMORANDUM  FOR  DOD  IG 

SUBJECT:  Air  Force  Comments  to  DOD  Draft  Audit  Report  On 
Lessons-Leamed  Programs  Within  the  Military 
Departments  (Project  No.  2  AG  -5007)  — INFORMATON 
MEMORANDUM 

The  attached  memorandum  is  the  Air  Force  Response  to  the 
subject  report. 

The  SAF/FMPF  point  of  contact  is  Mr.  Harvey  R  Morford, 
extension  7-6051. 


Liaison  and  Followup 
(Financial  Management) 


Atch 

Air  Force  response 
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Department  of  the  Air  Force  Comments 


DEPARTMENT  OF  THE  AIR  FORCE 


WASHINGTON  DC 


23  AUG  *1933 


OFFICE  OF  THE  ASSISTANT  SECRETART 


MEMORANDUM  FOR  SAF/FMP 

SUBJECT-  Air  Force  Response  to  DoD  IG  Draft  Report  on  Lessons-Learned  Programs 
Within  the  Military  Departments,  (Project  No.  2  AG-50()7)  -  ACTION 
MEMORANDUM 

We  agree  an  active  lessons-leamed  program  would  have  numerous  potential  benefits  to 
the  acquisition  process.  The  Air  Force  has  the  appropriate  systems  and  policies  in  place.  It  is 
an  effective  program.  In  addition  to  the  existing  Air  Force  Lessons-Learned  database,  HQ 
AFMC  has  developed  the  Air  Force  Acquisition  Model  (AFAM),  a  PC-based  application  that 
provides  information  for  Air  Force  weapon  systems  acquisition  tasks  across  all  functional 
disciplines.  AFAM  has  been  distributed  throughout  the  Air  Force  and  is  readily  available  for 
u.se.  Air  Force  policy  hUvS  been  enhanced  through  the  inclu.sion  of  lessons  learned 
requirements  in  the  approved  AF  Supplement  1/DoD  5000.2,  Defense  Acquisition 
Management  Policies  and  Procedures  (Atch  I ).  At  attachment  2  aie  our  specific  comments 

We  feel  that  the  Air  Foice  has  an  adequate  lessons-learned  system  in  place  and  is  taking 
steps  to  educate  its  program  managers  to  the  beneiils  of  these  programs.  As  with  all  efforts  of 
this  type,  lime  and  training  will  be  our  most  important  ally. 


8.  CMLDRESS.  Brffl  Gen.  USAF 


Attachments 

1.  Copy  AF  Supplement  J/DoDI  5000.2 

2.  Comments  on  Diall  DoD  IG  Repot  t 
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Department  of  the  Air  Force  Comments 


AF  Sup.  1/DoDI  5000.2 
Part  HE 
February  93 

PART  11 

SECTION  E  (Added)  (AF) 

AIR  FORCE  LESSONS  LEARNED 

References:  (a)  AF  Sup  1,  DoD  5000. 2 -M 

(Added)  (AF) 

1.  (Added)  (AF)  PURPOSE.  Establish  procedures  to  identify, 
document,  and  publish  acquisition  and  operational  lessons 
learned. 

2. d.  (Added)  (AF)  POLICY.  An  effective  program  to  share  both 

positive  and  negative  lessons  learned  is  of  significant  benefit 
to  the  Air  Force.  All  Air  Force  functional  activities  will 
provide  appropriate  lessons,  in  the  proper  format,  at  the  tine 
the  lessons  occur,  for  inclusion  in  the  lessons  learned  data 
bank.  Each  program  office  will  periodically  review  the 
appropriate  lessons  learned  data  bank  and  apply  the  applicable 
best  practices. 

3. C.  (Added)  (AF)  Submission  Procedures.  All  persons  involved 

with  accpiisition  and  operational  programs  are  responsible  for 
submitting  lessons  learned.  Submit  all  lessons  in  the  format 
described  in  AF  Sup  1,  DoD  5000. 2-M  to  ASC/CYM,  Wright-Pattcrson 
AFB,  OH  45433-5000. 
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Department  of  the  Air  Force  Comments 


AF  Sup.  1/DoDI  5000.2 
Part  HE 
February  93 

3. d.  (Added)  (AF)  Retrieval  Procedures.  To  retrieve  lessons 
from  the  data  bank  or  request  on-line  access,  forward  a  letter  to 
ASC/CYM. 

4. b.  (Added)  (AF)  RESPONSIBILITIES .  eSTT/AW;  the  system 

program  director  for  the  Air  Force  Lessons  Learned  Program,  will: 

(1)  conduct  an  annual  revalidation  of  lessons  contained 
in  the  data  bank. 

(2)  Publish  an  index  of  lessons  learned. 
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AF  Sup.  1/DoDI  5000.2 
Part  12B 
February  93 

PART  12 
SECTION  B 

JOINT/HULTISERVLCE  PROGRAMS 

References:  (c)  AFSC/AFLC/AMC/NMC  Memorandum  of  Agreement, 

(Added)  (AF)  20  July  1973,  on  the  Management  of 

Multi-Service  Systems/ Programs/Projects 

1,  (AF)  This  section  implements  the  Joint  Logistics  Commanders" 
Memorandum  of  Agreement  on  the  Management  of  Multi-Service 
Systems/Programs/ Projects,  reference  (c) . 

3.c.(5)  (Added)  (AF)  For  programs  for  which  the  Air  Force  is 
designated  the  lead  DoD  component,  the  Air  Force  will: 

(a)  Assign  the  SPD/PM. 

(b)  Establish  an  official  manning  document  for  the 
system  program  office  that  will  incorporate  the  positions  to  be 
occupied  by  representatives  of  the  participating  Services.  The 
manning  document  shall  designate  a  key  position  for  occupancy  by 
the  senior  representative  from  each  of  the  participating 
Services.  This  key  position  will  report  directly  to,  or  have 
direct  access  to,  the  SPD/PM.  This  key  position  could  include 
assignment  as  Deputy  System  Program  Director/ Program  Manager. 

The  Deputy  SPD/PM  will  function  as  the  participating  Service's 
representative,  with  responsibilities  and  authorities  as  outlined 
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Department  of  the  Air  Force  Comments 


COMMENTS 

ON 

CHANGES  TO  DoDI  5000.2 


Refeience  Page  10,  Part  II,  Consequences  and  Benefits,  Second  Paragraph  under 
Consequences:  The  slip  in  the  Navy's  AMR  A  AM  initial  operating  capability  (IOC)  date  is  not 
a  result  of  any  development  problems  with  the  missile,  but  rather  with  F/A-18  software 
problems.  The  Navy  only  recently  fielded  an  operational  flight  program  for  their  F/A-18  that 
could  support  AMRAAM.  Therefore,  the  slip  of  the  Navy  IOC  date  is  not  valid  consequence 
of  a  failure  to  use  a  lesson-learned  program. 

Reference  Page  13,  Recommendation  4:  Concur.  Air  Force  is  working  to  institute  a  set 
of  specific  processes  for  each  of  our  major  systems  acquisition  areas.  Our  single  manager 
concept  will  alleviate  many  of  these  problems.  The  implementing  document  for  DoDI  5000.2 
will  assist  in  enforcing  this  policy.  In  the  test  arena,  the  Single  Face  to  the  Customer  offices 
will  also  standardize  a  methodology  for  lest  program  design  and  execution.  As  a  result, 
lessons-learned  will  natui  ally  accumulate  and  improve  the  test  processes-thc  processes  will 
then  disseminate  the  lessons  to  other  test  programs 
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